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(57) ABSTRACT 

A system for rate control during transcoding and encoding of 
digital video data in a multi-program transmission environ- 
ment accommodates changes in the GOP structure of any of 
the programs, e.g., due to a commercial insertion or the like. 
A hierarchical bit allocation scheme is used, where bits are 
allocated on a super group of pictures (GOP) level, i.e., a 
grouping of multiple GOPs, a super frame level, and an 
individual frame level. The bit allocation for the super GOP 
and each picture is adjusted when a change in the structure 
of a group of pictures (GOP) is detected for any of the 
pre-compressed programs. The super GOP length may also 
be adjusted if required to maintain an integer number of 
GOPs therein. A change in the GOP structure is detected 
based on a change in the GOP length or sub_GOP length 
(distance between P pictures). For pre-compressed data, 
initial GOP lengths and sub_GOP lengths can be assumed 
and adjusted as required based on actual measurements. For 
a current I or P picture, the sub GOP length is calculated 
based on temporal references of the current picture and a 
previous I or P picture. For a current B picture, the same 
calculation is used, plus one picture. 

34 Claims, 7 Drawing Sheets 
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STATISTICAL MULTIPLEXER AND 
REMULTIPLEXER THAT ACCOMMODATES 
CHANGES IN STRUCTURE OF GROUP OF 
PICTURES 



BACKGROUND OF THE INVENTION 

The present invention relates to rate control during 
transcoding and encoding of digital video programs in a 
multi -program transmission environment, where several 
programs are multiplexed and transmitted over a single 
communication channel. 

Commonly, it is necessary to adjust a bit rate of digital 
video programs that are provided, e.g., to subscriber termi- 
nals in a cable television network or the like. For example, 
a first group of signals may be received at a headend via a 
satellite transmission. The headend operator may desire to 
forward selected programs to the subscribers while adding 
programs (e.g., commercials or other content) from a local 
source, such as storage media or a local live feed. 
Additionally, it is often necessary to provide the programs 
within an overall available channel bandwidth. It may also 
be desired to change the relative quality level of a program 
by allocating more or fewer bits. 

Accordingly, the statistical multiplexer (stat mux), or 
encoder, which includes a number of encoders for encoding 
uncompressed digital video signals at a specified bit rate, has 
been developed. The statistical remultiplexer (stat remux), 
or transcoder, which handles pre-compressed video bit 
streams by re-compressing them at a specified bit rate, has 
also been developed. Moreover, functions of a stat mux and 
remux may be combined when it is desired to transcode 
pre-compressed data while also coding uncompressed data 
for transport in a common output bitstream. Uncompressed 
programs are coded for the first time, while compressed 
programs are re-encoded, typically at a different bit rate. 

For MPEG applications, a stat mux/remux must accom- 
modate three different picture types (I, P and B), which 
usually require quite different numbers of bits because of the 
different nature of their temporal processing. 

Bit allocation strategies should take the picture types into 
consideration. For a given bit budget, this implies a require- 
ment of a priori knowledge of the picture organizations 
(GOP) of the programs. The requirement is not a problem for 
encoders because encoders can plan ahead for types and 
arrangement of pictures that it will output. However, such a 
priori knowledge is typically not available for transcoders, 
which deal with pre-compressed video bit streams. 

Accordingly, it would be desirable to provide a stat 
mux/remux system that removes the assumption that the 
picture organizations are available for all the programs. 

The system should provide a novel adaptive bit allocation 
strategy for a stat mux/remux system that requires no a priori 
(beforehand) knowledge of program picture organization. 
The system should be able to address any changes in 
program picture organizations, including a change in the 
GOP length and/or the sub-GOP length (the distance 
between two P-pictures in either encoding or display order). 

The system should include transcoders for processing 
pre-compressed video bit streams, or both encoders and 
transcoders, for additionally handling uncompressed digital 
video signals. 

Moreover, the system should accommodate changes in the 
group of pictures (GOP) structure of pre-compressed 
bitstreams, for example, due to switching channels, com- 
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mercial insertion, changes in the program content (e.g., due 
to a transition from a movie to a news announcement or a 
sports event), and the like. The system should handle struc- 
ture changes that occur at a GOP boundary or within a GOP. 

5 The system should accommodate changes in a GOP 
length and/or a sub_GOP length. 

The system should overcome difficulties in integrating 
pre-compressed program bit streams (that include, e.g., 
video materials such as films and the like that are pre- 

10 compressed and stored for subsequent transmission) with 
uncompressed digital video sequences. 

The system should not require a priori (beforehand) 
knowledge of the picture organization (GOP structure) of 
programs (e.g., the GOP length, and arrangement of different 

15 picture types in the GOP). The system should avoid a 
processing delay of about one GOP which would otherwise 
be incurred to extract the complete GOP structure informa- 
tion from a pre-compressed bit stream. The system should 
also avoid the need to store the data corresponding to the 

20 GOP, thereby reducing the memory size required for 
transcoding. 

The present invention provides a system having the above 
and other advantages. 

25 SUMMARY OF THE INVENTION 

The present invention relates to rate control during 
transcoding and coding of digital video data in a multi- 
program transmission environment. 

The invention accounts for changes in the structure of a 
GOP, e.g., due to a commercial insertion or the like, to adjust 
the bit allocation for each picture accordingly. A change in 
the GOP length (increase or decrease), and/or a sub_GOP 
length can be accommodated. For example, a change in 

35 picture sequence from IBBPBBPBB ... to IBPBPBPB . . . 
results in the sub_GOP changing from three to two. 

The sub_GOP length contains information about the 
specific picture types and their arrangement in a GOP. 
Hence, even when the GOP length remains the same during 

4Q a change in the input video source, we may have a different 
arrangement of picture types if the sub_GOP changes. 

Moreover, the invention operates with a hierarchical bit 
allocation scheme, where bits are allocated on a super GOP, 
super frame, and individual frame levels. 

45 The change in the GOP structure can occur anywhere, 
e.g., within a GOP or super GOP, or at a boundary of a GOP 
or super GOP. Once the GOP of a program changes, the 
length of the super GOP is re -calculated, and the allocation 
of bits to the super GOP is adjusted accordingly. 

50 If there is a switch during the middle of an old GOP to 
another program of a new GOP, this is treated as a transition 
from the old GOP to an old incomplete GOP, and then to the 
new GOP. 

A particular method in accordance with the invention is 
55 provided for coding a plurality L of video programs. The 
method includes the step of allocating bits for coding the 
video programs according to a hierarchical scheme that 
includes at least: (a) a super group of pictures (GOP) level, 
wherein a super GOP comprises at least one GOP from each 
60 of the video programs, and (b) an individual frame level. The 
allocating step is responsive to respective GOP structures of 
the video programs. The respective GOP structures of the 
video programs are monitored to detect any changes thereof. 
When a change in the GOP structure of at least one of the 
65 video programs is detected, the allocation of bits for coding 
the video programs in the super GOP is adjusted according 
to the change. 
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Accordingly, the invention can optimize the allocation of A stat mux has encoders, while a stat remux only needs 

bits since a rate control processor maintains updated knowl- transcoders, and a stat mux/remux has both encoders and 

edge of each picture type and the arrangement of pictures for transcoders. 

the different programs to the maximum extent possible. FIG. 1 illustrates a stat mux/remux system in accordance 

Corresponding apparatuses arc also presented. 5 with the present invention. The inputs to the stat mux/remux 

system 100 can include uncompressed digital video 

BRIEF DESCRIPTION OF THE DRAWINGS sequences and/or pre-compressed bit streams. The uncom- 

n „ ^ .„ . ... . pressed digital video sequences, e.g., programs 1 and 2 are 

FIG. 1 illustrates a stat mux/remux with both encoders £ ncoded b no and m respective iv, e .g., which 

and transcoders, and a joint rate control engine, in accor- 1Q u$c ^ Mp£G vkJeo coding slandafd ^ pre . comprcssed 

dance with the present invention. programs (e.g., bitstreams) L-l and L are processed, 

FIG. 2 illustrates a super group -of-pictures (GOP) con- respectively, by transcoders 140 and 150, GOP structure 

struct in accordance with the present invention. monitors 142 and 152, and optional bit rate functions 144 

FIG. 3 illustrates a super frame construct in accordance and 154. 

with the present invention. 15 Any known transcoder configuration may be used. 

FIG. 4 illustrates a decoder for receiving data from the stat The example GOP structure monitors 142 and 152 operate 

mux/remux of FIG. 1 in accordance with the present inven- as will be discussed in greater detail below to provide the 

t j 0D GOP lengths N L-1 and N L , and sub GOP lengths and 

FIG. 5 illustrates a video program with a change in the M ^> of , thc respective (L-l)th and Lth programs to the rate 

sub GOP length, and subsequent adjustment of the super 20 co £ tro1 Pressor 110, which adjusts the bit allocation to the 

GOP length, in accordance with the present invention. dlffcrcnt P ro S rams < via thc Ur S ct blt ratc ^ in rcs P onsc 

* • , , , thereto. 

FIG. 6 illustrates a video program with an increase in the ^ e k bit ratc fu nctions 144, 154 are optional and 

GOP length in accordance with the present invention. are used tQ monitQr ^ bU ^ of me pre . compressed input 

FIG. 7 illustrates a video program with a decrease in the 25 data f rames . As discussed further in connection with equa- 

GOP length in accordance with the present invention. tion 14, the rate control processor may reduce the target bit 

rate for a transcoder so that it does not exceed the measured 
bit rate. 

Note that the invention is operative when only pre- 

The present invention relates to rate control during 30 compressed bitstreams are present. Uncompressed program 

transcoding and coding of digital video data in a multi- data is accommodated, but is not required. Moreover, any 

program transmission environment number of pre-compressed bitstreams may be 

accommodated, although only two are shown in FIG. 1 for 

1. INTRODUCTION simplicity. 

With recent advances in digital video compression and 35 ^ transcoded data and encoded data (when present) are 

digital transmission, it is possible to deliver several video provided to a MUX 160 and buffer 170 prior to being 

programs in the same bandwidth presently occupied by a transmitted on a channel, typically to a decoder population, 

single analog TV channel. To be squeezed into a fixed-rate The channel may be part of a broadband communication 

channel, the video programs have to share the channel network, such as a cable or satellite television network, for 

capacity. Specifically, raw pixel data is compressed, and 40 exam pl e - 

compressed bits are transcoded. Encoders and transcoders The pre-compressed program data may be provided from 

can be thought of as rale-conversion engines. An encoder an V source > such ^ a stora 8 e raedium ( e -S-. magnetic tape or 

compresses a digital video sequence into a bit stream at a compact disc), or from a satellite transmission or stat mux, 

much lower rate while a transcoder converts a pre- for example. Pre-compressed data, such as for commercial 

compressed video bit stream into another bit stream at a new « insertion or insertion of local news or other programming, 

(low) rate. The aggregate rate of these compressed bit ma y be provided locally for use in conjunction with pre- 

streams, however, has to be equal to, or less than, the compressed data that is received from a remote source, 

channel rate. This can be achieved by controlling either each The buffer 170 a fullness level signal to a rate 

individual rate (independent coding) or the aggregate rate control processor 110, which may adjust the target bit 

(joint coding). 50 allocations accordingly to avoid a buffer overflow or under- 

In independent coding, rate control can only be performed ^ ow> . . ., , 

. 7. , - f „ Mm A user interface 108 may communicate with the rate 

across the time and spatial dimensions of a single program. A . ' ( _ AD 

1 * . a- » 1 v ,t_ *Z n control processor 110, for example, to set a GOP structure 

In joint coding, control is extended to the program dimen- . . r , ._ A ' A , . r . * , . . 

. / ft. \ ■ r ™ e^ a A nm :„ for the encoders 120, 130, to designate certain encoders or 

sion(e.g., multiple programs). This implies more freedom in _ . ' * ° , . 

u V. tU K U K , ' ™a 55 transcoders as having a higher pnonty, and so forth. 

allocating the channel capacity among programs and, a t . * . . , t - , c 

iU c & , 1 r ■ . n . ™t „u;~\l The stat mux/remux 100 provides a relatively uniform 

therefore, more control of picture quality among multiple ' *\ . 3 . 

tk - 0 r picture quality (or other designated quality level when 

programs as well as within a program. 7 v . . . b . 2 

r f . f. 1 different program priorities are assigned) within a program, 

Asystem that is able to put multiple video programs into and if necessarVy am0Qg programs . To achie ve this goal, 

a fixed-rate channel is called a: 6Q chanQcl ^ dly ^ dynarnically distributed among pro- 

1. stat mux if the inputs are all uncompressed digital video grams acC ording to a program priority as well as a frame 
signals, l cvc j pr0 g ram complexity measure. 

2. stat remux if the inputs are all pre-compressed video bit Each MPEG encoder 120, 130 or transcoder, 140, 150 
streams, or receives a target number of bits, Tj, T 2 , T^ and T L , 

3. stat mux/remux if the inputs are both uncompressed 65 respectively, from a rate control processor 110 at each frame, 
digital video signals and pre-compressed video bit The rate control processor 110 includes a super GOP level 
streams. processing function 106, a super frame level processing 
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function 104, a frame level processing function 102, and a 09/141,265 to L. Wang and A. Luthra, filed Aug. 27, 1998, 
complexity processor 105. These processing functions may and entitled "Dynamic Bit Allocation For Statistical Multi- 
share common hardware such as memory and processing plexing Of Compressed And Uncompressed Digital Video 
chips, but are shown individually for simplicity. Signals." At the top level of the hierarchy, the concept of a 

The target number of bits for each frame of a program is 5 super GOP (Groups of Pictures) is introduced. Specifically, 

met by adjusting the quantization parameter in the MPEG the programs are conceptually divided into super GOPs, 

encoder or transcoder. The resulting number of compressed each having the same number of I, P and B pictures, that are 

bits, R, as well as the average quantization parameter, Q, assigned the same nominal number of bits, 

used for each frame are then sent to the rate control A super frame is defined at a middle level of the hierarchy, 

processor 110 as feedback data. Specifically, the encoder 10 which is a collection of frames, one from each of the 

120, encoder 130, transcoder 140 and transcoder 150 pro- programs at the same frame instant. A super frame is 

duce Rj, R 2 , R L _! and K L bits, respectively, using quanti- assigned a target rate according to its relative complexity 

zation parameters Q lr Ql-i arJ d Ql» respectively. The measures. The same idea is extended to the regular, indi- 

complexity processor 105 calculates corresponding com- vidual frame, at a lower level of the hierarchy, where a target 

plexity values C using R and Q for each program. The rate 15 ra te for a frame is proportional to its complexity measure, 

control processor 110 then determines a new target number Moreover, to ensure the encoder and decoder buffers 

of bits for each new program frame or picture based on the nevcf ovecflow or underflow, and to limit each individual 

program complexity at the frame level. average rate, additional constraints are applied on the target 

The rate control processor 110 is also responsive to the number 0 f bits for the super frame as well as for the regular 

GOP structure data from the transcoders 140, 150. 20 f rame 

Ideally, a stat mux/remux system distributes .the channel ^ { ^ hierarchical d ic 

capacity over the input programs accordmg to die programs ^ mux/rcmux 

relative complexity measures. That is, more complex pro- GOP and Nominal Rate 

grams are assigned more bits, and ^^^V™*™* As shown in FIG. 2, we conceptually divide L input video 

are assigned fewer bits For a given ^ 25 .^.^ G P 0 p s (L,N) in terms of the 

that the rate assigned for a program d pends on the com P ^ ^ ^ gqP 

plexuy measures of the program and the other pro^ams^ ^ ^ ^ % 

Since the program complexity may vary with time, the ^ ^ £ * Each program 

programs' relate ; complexity measures vary as well. The I * ^ 2 Veont«ns a number of complete 

distribution of channel capacity, or bit allocation, over 30 fr f mes f ^ ^ or more G o Ps . A second super GOP 202 

programs therefore has to be a tnne-varying function (e.g., ^ ^ from program t pfogram % 

. - . ' . _ j .V , j • It \m — — u iooyi\ includes data frames from program 1 (214), program 2 

mg of moving pictures and associated audio , March 1994) *u™„k t 7 j jqa\ 

a,PandB),andth^^^^ £ 0 £ wfaich fc ^ tQ ^ 

can be very flexible. Due to the different nature of their ^ * ^ of ^ ^ GQp 

temporal processing, the three pictures may require very t ; * 

different numbers of bits. Hence, to wisely use the bit budget ien S ins or • • • > L > ie » 

over the input programs, the program picture organizations 40 n-lcm{N x , . . . , tfj (i) 

should also be taken into consideration. However, for a .^n, r *t 

transcoder, the program picture organization is embedded in For example, if there are two different GOP lengths for N 

the pre-compressed bit stream and is therefore not available programs, say nine and fifteen, the super GOP length N-45. 

as the bit stream is being received. While a transcoder can Since N is the smallest integer that can be divided by all the 

learn the program picture organization of an input pre- 45 program GOP lengths, N„M, 2, L, the super GOP^L, 

compressed bit stream, e.g., by scanning the bit stream by N) are the smallest identical groups containing the same 

about one GOP, this results in processing delays and the rmmber of frames of each picture type. A super GOPC^O 

need for additional memory space, which increases costs. already contains the integer number of GOPs for each 

Moreover, a further complication is that the extra memory Program 1, that is, N/N;. Super GOPs(L,N) with N defined in 

requirement will vary for programs with different GOP- 50 0) contain the same number of I, P and B pictures and 

lengths hence, they are assigned the same nominal number of bits, 

Trie present invention operates on the assumption that a ^^(txAO^*.^^/) ( 2 ) 

priori knowledge of program GOP structures is not available ' 

to the transcoders. The invention starts with a reasonable where R^^^a is the average number of bits per frame, 

assumed program picture organizations (GOP length and 55 The number of I, P and B pictures in a GOP for program 

sub_GOP length), and provides gradual adjustments to the 1 are, respectively, N^l, N^-N/Mj-N^-Nz/M,-!, and 

assumed organization when necessary. A GOP length of N^-ty-N^-N^-ty-N^-l. 

fifteen pictures, and a sub GOP length of three pictures is a 2.2 Super Frame and Target Rate 

reasonable assumed organization in many cases. For FIG. 3 illustrates a super frame, which is a collection of 

example, N«15 and M»3 corresponds to a picture organi- 60 L frames, one from each of L programs at the same time 

zation of . . . BBIBBPBBPBBPBBPBBI . . . , while N=10 instant. 

and M-2 corresponds to a picture organization of . . . At each frame instant n, we can further imagine a super 

BIBPBPBPBPBI . . . frame 300, which is a collection of L frames or pictures, one 

from each of the L programs taken at the same frame instant. 

2. DYNAMIC BIT ALLOCATION 65 For examp ie f f rame 310 from program 1, frame 320 from 

A hierarchical dynamic bit allocation strategy for a stat program 2, . . , , through frame 330 of program L are 

mux system is discussed in U.S. patent application Ser. No. conceptually arranged in the super frame 300. Clearly, a 
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super GOP consists of N super frames. Moreover, since 2.4 Target Rate for Regular Frame 

these L programs may have different GOP structures, the L Given a target number of bits for a super frame n, T M , the 

frames in a super frame can have different picture types. target number of bits for frame n of program 1 w 
The target number of bits for super frame n is given as: 

5 „ *ifim»&M (4) 

Tin ~ Tn- 



L (3) - „ 



£ [nuatficCu ^^^AwQil iq HerCj the numerator on the righl is lhe complexity mea- 

sure for frame n of program 1, and the denominator can be 
considered as the complexity measure for super frame n. 

where C w is the complexity measure for frame n of HcncCj a framc is a largct ratc proportional to its 

program 1 with picture type te{I,P,B}, and it can be either comp i exit y measure. 

C /(/> C^ or C,^, depending upon the associated picture type ^ tafgcl ratcs dctcrmincd by equation (4) are based 

of I, P or B.TM5 defines a complexity measure for a frame upon the frame relative complexity measure. For 

as the product of the number of bits generated from the transcoders, they may need further adjustment. Specifically, 

frame and the average quantization parameter used for the the tafgel rate for a frame determined by (4) could be larger, 

frame. Other complexity measures are also allowed. or &mdRcTf than me old rate for me f rame in the input 

a, is a constant factor for addressing the spatial resolution ^ pre -compressed bit stream. A rate-conversion transcoder is, 

of program 1. however, a device for converting a pre-compressed video bit 

p ; is the quality (or priority) weighting factor for program stream into anotner bit stream at a new rate. It cannot 

1, which will be determined by the program provider. improve the quality of a pre-compressed video signal 

y w is a constant factor used to compensate for the picture regar dl ess 0 f whether the new bit rate is smaller or greater 

type t € {I,P,B} of frame n of program 1. It can be either y» y P lhan me old bU rale 

or Yb, depending upon the picture type of I, P or B. -Rating m0 re bits for a (pre-compressed) frame 

n u , riff and n lwB are the remaining number of I, P and B waste s the bits. Hence, in accordance with the invention, if 

pictures, respectively, for program 1 in the current super tfac targel rate fof framc n of program s 1, T ljB , (4) is greater 

GOP. For a new super GOP, they are reset, and then ^ thfi oW fate for the frame k me input pre-compressed 

decreased by one after a picture of the corresponding type is bit strcam> ^ y the old rate should be applied, i.e., 
processed. 

R is the remaining number of bits for the current super ( r if r < R (5) 

G0P ' defined as T "-L»,»i«- 

L 

H ~ * + Yj (ru_1 " Rt "-i )• 35 This can be achieved using the example bit rate functions 

/=l 144, 154 of FIG. 1 by measuring the bit rate of a picture in 

the received bit stream, e.g., using a bit counter, and sending 

Here, T, j and R^_ 1 are, respectively, the target and actual this information to the rate control processor 110 to reduce 

rates foArame n-1 of program 1. Hence, 4Q the allocated bits, T, for the associated transcoder, if 

required. 

i 2.5 Preventing Decoder Buffer Overflow or Underflow 

Y t^-i - tfot-i ) FIG. 4 illustrates a decoder for receiving data from the stat 

'=i mux/remux of FIG. 1, where packets of a selected program 

are extracted and decoded. At the receiving end 480, the 

is the number of leftover bits from frames n-1 of program " decoder 495 is allowed to select the desired program and 

MA . ... L, which can be either positive or negative. At &r™ lU ?* x) ,* e , ^P 0 ^ P a f ets 

the Uginning of a new super GOP, R needs to be updated as *mux 48S. The de-mult.plexed lb. s ream b at a vanable 

R-R+L* where R on the right side of the equation is the »««• & »*• encoder^ respons.bihty to ensure the decoder 

number ofbits leftover from the previous super GOP, which 50 490 never overflows or underflows. Addmonal con- 

„„„ u„ „r «^r, Q t,w straints are therefore applied at the encoder, 

car i be positive or nega ive. This can be achieved using the techniques of the afore- 

Note that the numerator on the right side of equation (3) 11 . * "yl " # ,. b c m no/«i ach 

r , c 11 it c ' mentioned U.S. patent application Ser. No. 09/553,457 to 

is the sum of complexity measures for all the frames in super ^ y y , 

« ui* ^uiuui v u F j " app l y a constraint on the target rate for each regular frame, 

frame n. On the other hand, the denominator can be con- * y rL . ■ \ \a a \a 'n 

sidered as the sum of complexity measures for all the 55 2 ' 6 „ 7 CoDSlra ; nt on ™™ and Min Rate 

. r . „ t * or rr\o u,„„ ^..Mion We can also control the average bit rate over a certain 

remaining frames in the current super GOP. Hence, equation * for 

3) actually assigns a target number of bits for a super frame * * * be achieved 

m proportion to the super frame's complexity measure. P aforementioned U.S. patent 

2.3. Preventing Encoder Buffer Overflow or Underflow T lication Ser No 09/553 457 

To prevent the encoder buffer 170 from overflowing or 60 a ^ C 1 * ' 

underflowing, certain constraints are applied by setting 3 CHANGES IN PROGRAM GOPS 
upper and lower bounds for the target rate for super frames. 

This can be achieved according to the techniques set forth in The program GOP structure (including GOP length, type 

U.S. patent application Ser. No 09/533,457 to L. Wang, filed and arrangement of pictures, and sub GOP length) plays an 

Apr, 19, 2000, and entitled "Rate Control And Buffer 65 important role in determining a target rate for a frame, as 

Protection For Variable Bit Rate Video Programs Over A shown in equations (3,4). However, a program GOP struc- 

Constant Rate Channel". ture may vary from time to time. For example, for a 
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transcoder, the input video bit stream may consist of seg- at 520, when a new super GOP length, 

ments of different materials/content, such as films, news, Let tem P-™f-P tc * and * m P-™f-P ic2 be the temporal references 

sports, music, etc., that are pre -encoded with different GOP for the previous I or P picture and the current picture for 

structures. Another example is commercial insertion, where program 1, respectively. 

the inserted commercial may be pre-coded with a different $ If the previous picture is an I or P picture, and the current 

GOP structure than the original program to be inserted. picturc ^ a B p i ctur c, the B picture should be displayed 

In accordance with the invention, the bit allocation adapts before the I or P (in display order). Thus, the B picture has 

to changes in the program GOP structures, e.g., by adjusting a smaller temporal reference. Now, if the current picture is 

the size of the super GOP and other parameters for deter- an I or P picture, the actual sub_GOP length for program 1 

mining the target rates of frames. 10 is; 
3.1 Program sub_GOP Length 

The following shows a general form of picture organiza- Af /t( , CM(fl r^mp_ref_pic2-temp_ref_picl. (6) 

tion for an example program 1. The first row is in display ^ t . .„ , „ . 

order and the second row is the corresponding encoding Otherwise, if the current picture is a B picture, 

° rden Tp p RPR 15 M / , oe(M/ -t C mp_r e f_picl-t C mp_r e f_pic2+l (7) 

P m +2M t • • • ^m+z// • * • > No sub_GOP is determined for a current picture whose 

• • • I«B I 1. WI xp^wi • • ■ B-i^WW! • • • P revious P^ture is a B-picture. The start of the current 

R m +M-iPm+2M * - * ^4jv • picture in FIG. 5, which is an I picture and the first picture 

HereTthe subscripts are" the picture's temporal reference, 20 of the GOP 552, is at a time 515. The start of the next picture 

and N, and M, are the GOP length and sub_GOP length, is at a time 520. 

respectively. The pictures in the compressed video bit stream Clearly, if M^M, actuah the current sub„GOP for pro- 

that are input to a transcoder are in encoding order. The gram 1 is correct. Hence, no further action is needed, 

display order is recovered at the decoder. However, if M,*M /jflCfHO/ , we set M r M itaciual , and adjust 

For an encoder that encodes uncompressed video data, the 25 other parameters as follows: 

changes in sub_GOP length of a program are available since 1 . First, the program GOP length, N„ has to be divisible 

the GOP program structure is available, thus allowing the by the new M, (i.e., integrally divisible— with no 

encoder to plan ahead for bit allocation. remainder). If not, we adjust it by 

However, for a transcoder, the changes in sub_GOP 

structures are embedded in the input p re-compressed bit 30 



Nf{NJMftM t (8) 

stream. In accordance with the invention, the sub_GOP where ry] denotes integer division with truncation of the 

length of a program is calculated at each current frame/ result toward zero. 

picture (I, P or B) if the previous frame is an I or a P picture For example, assume N,=15 and M=3 initially. Then, M 

by checking the temporal references of the current picture changes to M-2. Using eqn, 8, the adjusted program GOP 

and the previous I or P picture. If the previous frame is a 35 i en g| n N / «14. 

B-picture, we do not check the sub_GOP. In MPEG syntax, 2 . Second, if N ; changes, we reset the super GOP length 

the temporal reference of a picture is included in the picture Dy e q Uat i 0 n (I). The new super GOP 540 now starts 

header that is located at the beginning of the segment of right at the following picture, at time 520. Note that the 

compressed bits for the picture. Specifically, the term program GOPs do not have to be aligned with the new super 

"temporal_reference" is a 10-bit unsigned integer associ- 40 G0Ps ^ num ber of 1, P and B pictures in the (new) super 

ated with each input picture. It is incremented by one, G0P 54 o can also be calculated with the new super GOP 

modulo 1024, for each input frame. For example, temporal_ length> ^ y N ^ as ^ respectively, n^N^N/N^N/N,, 

reference may have values of 67, 68, 69 and so forth. Hence, n / ^=N J ^*(N/N / )=N/M r n />/) and n / ^=N J ^*(N/N / )=N-n / ^- 

a transcoder (or, more specifically, the associated GOP n 

structure monitors 142 or 152 of FIG. 1) can access the 45 3. Finally, the nominal number of bits for the (new) super 

temporal reference of a current picture before the transcoder qqp ^ now equal to: 
actually processes (transcodes) the picture. The temporal 

reference of the previous I or P picture is already available ^^^-(ix^^)^^^. (9) 

because the picture has been processed. m , - . . - , , x ~ rtri „ tr , 

FIG. 5 illustrates a video program with a change in the so ™» number of bits for the first (new) super GOP 540, 

sub_GOP length, and subsequent adjustment of the super however, rnay be slightly different than for other subsequent 

GOP length, in accordance with the present invention. ™P™ G° Ps thc u "P" 3°, "2* I , 

Note that the program GOPs for the different programs previous super GOP 530 (with length U oM ) is an unfinished 

1, . . . , Ldo not necessarily align with one another. super GOP that has leftover bits that were previously 

An example program 1 that is transcoded includes GOPs 55 ^located, but not yet used In accordance with the mvcnUon, 

546, 548 and 550 having a first GOP structure (GOP length thesc blts can bc ^located to the first new super GOP 540. 

and/or sub„GOP length), and GOPs 552, 554 and 556 Specifically, let N^ rf be the old super GOP length and N be 

having a second, different GOP structure. Note that the GOP the nu _ mber of frame f °1_ P ro & ram 1 m the ^ su P c u r G ° P th L at 

structure can change for more than one program at the same have been processed. The remaining number of bits for the 

time. The super GOP length and bit budget are adjusted 60 first new su P er G0P K then: 

accordingly. , . /wr^W(T taiW HV] (10) 

Moreover, as mentioned, the programs may comprise 

transcoded data only, or both transcoded and encoded data. Here, T LxN . is the nominal number of bits for LxN' frames 

At a time indicated at 510, an old super GOP 530 with and (T iKjv tf -R) is the number of bits used for the frames in 

length, N oW , is determined. A sub___GOP (the distance 65 the old (unfinished) super GOP 530. Hence, [T^^- 

between two P-pictures in either encoding or display order) (T txA? -R)] denotes the leftover bits from the old 

for each program does not change up until a time indicated (unfinished) super GOP 530. 
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With the updated information, we can determine the target 
rates for the current and future frames (eqns. 3 and 4). Note 

that the sub GOP length of a program can be verified and/or 

corrected at each frame if the previous frame is an I or a P 
picture, and if necessary, other parameters are also adjusted 
accordingly (eqns. 8-10). 
3.2 Program GOP Length 

Let N/,<>w and be the GOP lengths of the previous 
and the current GOPs for program 1, respectively. There are 
three possible cases regarding the GOP lengths, i.e., 



10 



(11) 



For encoders, both N told and N /M<r>v are available. 
However, for a transcoder, the information on GOP length is 
embedded in the input pre-compressed bit stream. Before 
completely processing the current GOP of program 1, N /)WrKJ 
is not available. 

We set the current GOP length N,=N, >£>W for the 
transcoder. Clearly, if N, ntfVV =N / oM , we have a correct GOP 
length for program 1. Hence, no further action is needed, 

FIG. 6 shows an example where the new GOP length for 
the current GOP is longer than the currendy used GOP 
length. 

Program 1 includes GOPs 646, 648 and 652 having a first 
GOP structure, and GOPs 654, 656 and 658 having a second, 
different GOP structure. 

An old super GOP 630 starts at a time 610, while a new 
super GOP 640 starts at a time 620. Old GOPs extend, e.g., 
from a time 632 to a time 635, and from a time 635 to a time 
620, A new GOP extends from a time 635 to a time 650. 

If N; ,^w>N,=N w ^ i.e., the actual (new) program GOP 
length is longer than the currently used (old) GOP length. 
The GOP structure monitor (142, 152) expects to reach a I 
picture at the end of the current GOP of N, frames, at time 
620. However, the end of a program GOP is not actually 
reached at time 620. In accordance with the invention, this 
alerts the GOP structure monitor of a change in the GOP 
structure. Specifically, it can be concluded that the actual 
GOP length is longer than the currently-assumed GOP 
length. 

Moreover, because of the nature of the program GOP 
structure (FIG. 2), the new GOP length will be at least M, 
(one sub GOP length) additional pictures longer. Here, we 
have assumed that the sub-GOP length M does not change 
within a GOP. We extend the current GOP of N, pictures by 
M; additional pictures, i.e., 



(13) 
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With the new N,, we recalculate the super GOP length by 
equation (1), and the nominal number of bits for the first new 
super GOP (and the following super GOPs) by equation (2). 
The first new super GOP 640 is aligned with the current P 
picture of program 1, as shown at time 620. Because of the 
leftover bits from the previous super GOP 630, the number 
of bits for the first new super GOP 640 is re-allocated as: 



55 



60 



12 



FIG. 7 shows an example where the new GOP length for 
the current GOP is shorter than the currently used GOP 
length. 

Program 1 includes GOPs 746, 748 and 752 having a first 
GOP structure, and GOPs 754, 756, 758 and 760 having a 
second, different GOP structure. 

An old super GOP 730 starts at a time 710, while a new 
super GOP 740 starts at a time 720. An old GOP extends 
from a time 732 to a time 735, and from a time 735 to a time 
750. A new GOP extends from a time 735 to a time 720. 

If N ; „ ew <N / -N, o/d , i.e., the actual GOP length is shorter 
than the currently used GOP length, an I picture (which 
denotes the start of a new GOP — there is only one I picture 
in a GOP) will be reached (at time 720) before the end of the 
current GOP of N, frames. At this point, the GOP structure 
monitor knows the actual GOP length, N, ttiewt since the GOP 
754 has just concluded. We set N^N^J W . Moreover, with 
the new program GOP length, N, we reset the super GOP 
length by (1), and recalculate the nominal number of bits for 
the new super GOPs by (2). Again, because of the leftover 
bits from the previous (unfinished) super GOP 730, the 
number of bits for the first new super GOP 740 may be 
slightly different than for subsequent super GOPs, that is, 



25 



where N oW and N w>v are the old and new super GOP lengths, 
respectively, and N' is the number of frames of program 1 in 
the old super GOP 630 that have been processed. If the 
extended GOP for program 1 is still shorter than the actual 
GOP, we repeat the above procedure (e.g., increasing N, by 
eqn. 12) until reaching the end of the actual GOP. 



(14) 



where N oW and N„ e>v are the old and new super GOP lengths, 
respectively, and N' is the number of frames of program 1 in 
the old super GOP 730 that have been processed. 

30 4. RATE CONTROL 

Once the target rate for a frame of a program is set, the 
next step is to achieve this rate, e.g., by adjusting the coding 
parameters, such as the quantization parameter, in the encod- 

35 ers and transcoders of the stat mux/remux system. Any 
suitable rate control scheme, such as those for use with the 
MPEG encoder, can be used. Some of them can be applied 
to the transcoders as well. 

Preferably, joint rate control is used to dynamically dis- 

40 tribute the channel capacity among the programs according 
to the programs' relative complexity measures. 

5. CONCLUSION 

A stat remux system is presented having encoders and 
transcoders that can handle both uncompressed digital video 
signals and pre-compressed video bit streams. The system 
implements a novel adaptive rate control, which dynami- 
cally distributes the channel capacity over the input video 
programs, either uncompressed digital video signals or 
pre-compressed video bit streams, on a frame-by-frame 
basis. The bit allocation strategy adopted in the adaptive rate 
control is able to address changes in picture organization 
(e.g., changes in GOP length or sub_GOP length) in the 
input pre-compressed bit streams. When such a change is 
detected, the bit allocation is recalculated. 

Moreover, hierarchical bit allocation is provided on a 
super GOP level, then on a super frame level, and finally at 
an individual frame level. 

Although the invention has been described in connection 
with various specific implementations, it should be appre- 
ciated that various adaptations and modifications can be 
made thereto without departing from the scope of the 
invention as set forth in the claims. 
65 What is claimed is: 

1. A method for transcoding a plurality L of pre- 
compressed video programs, comprising the steps of: 
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estimating a respective GOP structure for each video 
program; 

allocating bits for transcoding the pre-compressed video 
programs according to a hierarchical scheme that 
includes at least: (a) a super group of pictures (GOP) 
level, wherein a super GOP comprises at least one GOP 
from each of the video programs, and (b) an individual 
frame level; 

wherein said allocating step is initially responsive to said of: 
estimated respective GOP structures of the video pro- 
grams; 

monitoring the actual respective GOP structures of the 
pre-compressed video programs to detect any changes 
thereof; and 

when a change in the GOP structure of at least one of the 
video programs is detected, adjusting the allocation of 
bits for transcoding the video programs in the super 
GOP according to said change. 0 f: 

2. The method of claim 1, wherein: 
said monitoring step monitors the GOP structure of the 

video programs on a frame-by-frame basis. 

3. The method of claim 1, wherein: 
the change in GOP structure is due to a change in a video 

source. 25 

4. The method of claim 1, wherein: 
the change in GOP structure is due to a commercial 

insertion in the associated video program. 

5. The method of claim 1, wherein: 

an aggregate of the transmission rates for the video 30 
programs is maintained within a total available channel 
rate. 

6. The method of claim 1, wherein: 
the video programs are communicated over a broadband 

communication network to a decoder population. 

7. The method of claim 1, wherein: 
the hierarchical scheme includes a super frame level 

between the super GOP level and the frame level; and 
each super frame comprises L pictures at a common frame 
instant, one from each of the video programs. 

8. The method of claim 1, wherein: 
the L video programs have respective variable bit rates. 

9. The method of claim 1, comprising the further step of: 
determining whether a target bit rate that is set in said 

allocating step for a frame of pre-compressed data of at 
least one of the video programs is greater than a 
measured bit rate of the frame and, if so, reducing the 
target bit rate to the measured bit rate. 

10. The method of claim 1, wherein: 
the plurality of video programs further comprise at least 

one video program having uncompressed video data. 

11. The method of claim 1, wherein: 
the GOP structure comprises a program GOP length. 

12. The method of claim 11, comprising the further step 

of: 

starting a new super GOP, and allocating bits thereto, 
when a change in the length of a program GOP of at 
least one of the video programs is detected. 

13. The method of claim 11, wherein said estimating of a 
respective GOP structure for each video program comprises 
providing an assumed value for the program GOP length 
before a measurement of the program GOP length can be 
made; 

said method comprising the further steps of: 65 
measuring the program GOP length when the measure- 
ment can be made; and 



comparing the measured program GOP length to said 
value and, if there is a difference, setting said value 
to the measured program GOP length. 

14. The method of claim 12, wherein: 
the allocation of bits to the new super GOP accounts for 

unused bits from a previous super GOP that were 
previously allocated thereto but not used. 

15. The method of claim 12, comprising the further step 
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calculating a length of the new super GOP based on the 
changed program GOP length. 

16. The method of claim 14, wherein: 
the unused bits are determined based on a number of 

frames processed in the previous super GOP. 

17. The method of claim 1, wherein: 
the GOP structure comprises a sub GOP length. 

18. The method of claim 17, comprising the further step 



starting a new super GOP, and allocating bits thereto, 
when a change in the length of a sub GOP of at least one 
of the video programs is detected. 

19. The method of claim 17, wherein: 
the sub GOP length is calculated after bits are allocated in 

said allocating step to an I picture. 

20. The method of claim 17, wherein: 
the sub GOP length is associated with a distance between 

one of successive I-pictures or successive P-pictures in 
a program GOP. 

21. The method of claim 17, wherein: 
when a current picture of one of the video programs is an 

I or P picture, and a previous picture is an I or P picture, 
the sub GOP length is determined according to a 
difference between temporal references of the current 
picture and the previous picture. 

22. The method of claim 17, wherein: 
when a current picture of one of the video programs is a 

B picture, and a previous picture is an I or P picture, the 
sub GOP length is determined according to a difference 
between temporal references of the current picture and 
the previous picture, plus one picture. 

23. The method of claim 17, comprising the further step 
of: 

calculating the sub GOP length after processing an I or P 
picture. 

24. The method of claim 17, wherein said estimating of a 
respective GOP structure for each video program comprises 
providing an assumed value for the sub GOP length before 
a measurement of the sub GOP length can be made; 

said method comprising the further steps of: 

measuring the sub GOP length when the measurement 

can be made; and 
comparing the measured sub GOP length to said value 
and, if there is a difference, setting said value to the 
measured sub GOP length. 

25. The method of claim 17, wherein: 
when a change in the sub GOP length is detected, calcu- 
lating an associated new program GOP length. 

26. The method of claim 25, wherein: 
the new program GOP length N /jW>v is calculated accord- 
ing to N^ w ^N w yM,^J*M 

where N t old is the previous program GOP length; 



M 



l Stew 



is the new sub GOP length; and 



"T denotes integer division with truncation of the result 
toward zero. 
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27. The method of claim 25, wherein: 

after the new program GOP length is calculated, calcu- 
lating a new super GOP length in accordance therewith, 
and allocating bits to the new super GOP accordingly. 

28. The method of claim 1, wherein: s 
the GOP structure denotes picture types and their arrange- 
ment in a program GOP. 

29. The method of claim 1, wherein: 

a length of the super GOP is determined according to a 1Q 
least common multiple of the program GOP lengths of 
the respective video programs. 

30. An apparatus for transcoding a plurality L of pre- 
compressed video programs, comprising: 

means for inputting an estimated respective GOP struc- 15 
ture for each video program; 

means for allocating bits for transcoding the pre- 
compressed video programs according to a hierarchical 
scheme that includes at least: (a) a super group of 
pictures (GOP) level, wherein a super GOP comprises 20 
at least one GOP from each of the video programs, and 
(b) an individual frame level; 

wherein said allocating means is initially responsive to 
said estimated respective GOP structures of the video 
programs; 25 

means for monitoring the actual respective GOP struc- 
tures of the pre -compressed video programs to detect 
any changes thereof; and 

means for adjusting the allocation of bits for transcoding 3Q 
the video programs in the super GOP, when a change in 
the GOP structure of at least one of the video programs 
is detected, according to said change. 

31. An apparatus for transcoding a plurality L of pre- 
compressed video programs, comprising: 35 

a user interface for inputting an estimated respective GOP 
structure for each video program; 

at least two transcoders for receiving pre -compressed data 
of corresponding ones of the pre-compressed video 
programs; 40 

a rate control processor for allocating bits for transcoding 
the video programs according to a hierarchical scheme 
that includes at least: (a) a super group of pictures 
(GOP) level, wherein a super GOP comprises at least 
one GOP from each of the video programs, and (b) an 45 
individual frame level; 



wherein said rate control processor is initially responsive 
to said estimated respective GOP structures of the video 
programs; and 

a GOP structure monitor associated with each of the 
transcoders for monitoring the actual respective GOP 
structures of the associated pre-compressed video pro- 
grams to detect any changes thereof; 

wherein said rate control processor adjusts the allocation 
of bits for transcoding the video programs in the super 
GOP, when a change in the GOP structure of at least 
one of the video programs is detected, according to said 
change. 

32. A method in accordance with claim 1, comprising the 
further steps of: 

transcoding of said pre-compressed video programs; and 
encoding a plurality of uncompressed video programs; 
wherein said allocating step includes allocating bits for 
encoding of said uncompressed video programs in 
addition to allocating bits for transcoding of said pre- 
compressed video programs based on said hierarchical 
scheme. 

33. An apparatus in accordance with claim 30, further 
comprising: 

means for transcoding said pre-compressed video pro- 
grams; and 

means for encoding a plurality of uncompressed video 
programs; 

wherein said allocating means allocates bits for encoding 
of said uncompressed video programs in addition to 
allocating bits for transcoding of said pre-compressed 
video programs based on said hierarchical scheme. 

34. An apparatus in accordance with claim 32, further 
comprising: 

at least one encoder for receiving uncompressed video 
programs; 

wherein said rate control processor allocates bits for 
encoding of said uncompressed video programs in 
addition to allocating bits for transcoding of said pre- 
compressed video programs based on said hierarchical 
scheme. 



03/17/2004, EAST Version: 1.4.1 



UNITED STATES PATENT AND TRADEMARK OFFICE 

CERTIFICATE OF CORRECTION 



PATENT NO. : 6,643,327 Bl Page 1 of 1 

DATED : November 4, 2003 

INVENTOR(S) :Wang 



It is certified that error appears in the above-identified patent and that said Letters Patent is 
hereby corrected as shown below: 



Column 16, 

Line 35, change "32" to - 31 --. 



Signed and Sealed this 
Twentieth Day of January, 2004 



JON W. DUD AS 
Acting Director of the United States Patent and Trademark Office 



03/17/2004, EAST Version: 1.4.1 



